{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "title:generic"
   },
   "source": [
    "# Vertex AI Pipelines: Pipelines introduction for KFP\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "overview:pipelines,intro"
   },
   "source": [
    "## Overview\n",
    "\n",
    "This notebook provides an introduction to using [Vertex AI Pipelines](https://cloud.google.com/vertex-ai/docs/pipelines) with [the Kubeflow Pipelines (KFP) SDK](https://www.kubeflow.org/docs/components/pipelines/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "objective:pipelines,intro"
   },
   "source": [
    "### Learning objectives\n",
    "\n",
    "1. Initialize Vertex AI SDK for Python.\n",
    "2. Compile the pipeline.\n",
    "3. Run the pipeline.\n",
    "4. Specify a service account to use for a pipeline run.\n",
    "5. Use the Pipelines REST API."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "restart"
   },
   "source": [
    "Each learning objective will correspond to a __#TODO__ in this student lab notebook -- try to complete this notebook first and then review the [solution notebook](../solutions/pipelines_intro_kfp.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "install_aip:mbsdk"
   },
   "source": [
    "## Installation\n",
    "\n",
    "Install the packages required for executing this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "F_jyqAW_2Ouu"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33m  WARNING: The script tabulate is installed in '/home/jupyter/.local/bin' which is not on PATH.\n",
      "  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.\u001b[0m\u001b[33m\n",
      "\u001b[0m\u001b[33m  WARNING: The script strip-hints is installed in '/home/jupyter/.local/bin' which is not on PATH.\n",
      "  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.\u001b[0m\u001b[33m\n",
      "\u001b[0m\u001b[33m  WARNING: The script jsonschema is installed in '/home/jupyter/.local/bin' which is not on PATH.\n",
      "  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.\u001b[0m\u001b[33m\n",
      "\u001b[0m\u001b[33m  WARNING: The scripts dsl-compile, dsl-compile-v2 and kfp are installed in '/home/jupyter/.local/bin' which is not on PATH.\n",
      "  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.\u001b[0m\u001b[33m\n",
      "\u001b[0m\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
      "tensorflow-io 0.21.0 requires tensorflow-io-gcs-filesystem==0.21.0, which is not installed.\n",
      "tfx-bsl 1.10.1 requires pyarrow<7,>=6, but you have pyarrow 9.0.0 which is incompatible.\n",
      "tfx-bsl 1.10.1 requires tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,!=2.8.*,<3,>=1.15.5, but you have tensorflow 2.6.5 which is incompatible.\n",
      "tensorflow 2.6.5 requires typing-extensions<3.11,>=3.7, but you have typing-extensions 4.3.0 which is incompatible.\n",
      "tensorflow-transform 1.10.1 requires pyarrow<7,>=6, but you have pyarrow 9.0.0 which is incompatible.\n",
      "tensorflow-transform 1.10.1 requires tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,!=2.8.*,<2.10,>=1.15.5, but you have tensorflow 2.6.5 which is incompatible.\n",
      "tensorflow-serving-api 2.10.0 requires tensorflow<3,>=2.10.0, but you have tensorflow 2.6.5 which is incompatible.\n",
      "pandas-profiling 3.0.0 requires tangled-up-in-unicode==0.1.0, but you have tangled-up-in-unicode 0.2.0 which is incompatible.\n",
      "cloud-tpu-client 0.10 requires google-api-python-client==1.8.0, but you have google-api-python-client 1.12.11 which is incompatible.\n",
      "apache-beam 2.41.0 requires dill<0.3.2,>=0.3.1.1, but you have dill 0.3.5.1 which is incompatible.\n",
      "apache-beam 2.41.0 requires pyarrow<8.0.0,>=0.15.1, but you have pyarrow 9.0.0 which is incompatible.\u001b[0m\u001b[31m\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "import os\n",
    "\n",
    "# The Vertex AI Workbench Notebook product has specific requirements\n",
    "IS_WORKBENCH_NOTEBOOK = os.getenv(\"DL_ANACONDA_HOME\") and not os.getenv(\"VIRTUAL_ENV\")\n",
    "IS_USER_MANAGED_WORKBENCH_NOTEBOOK = os.path.exists(\n",
    "    \"/opt/deeplearning/metadata/env_version\"\n",
    ")\n",
    "\n",
    "# Vertex AI Notebook requires dependencies to be installed with '--user'\n",
    "USER_FLAG = \"\"\n",
    "if IS_WORKBENCH_NOTEBOOK:\n",
    "    USER_FLAG = \"--user\"\n",
    "\n",
    "! pip3 install --upgrade google-cloud-aiplatform {USER_FLAG} -q\n",
    "! pip3 install -U google-cloud-storage {USER_FLAG} -q\n",
    "! pip3 install {USER_FLAG} kfp google-cloud-pipeline-components --upgrade -q\n", 
    "! pip install kfp==1.8.14" 
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "restart"
   },
   "source": [
    "### Restart the kernel\n",
    "\n",
    "Once you've installed the additional packages, you need to restart the notebook kernel so it can find the packages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "lr4jnWxh2Oux"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "if not os.getenv(\"IS_TESTING\"):\n",
    "    # Automatically restart kernel after installs\n",
    "    import IPython\n",
    "\n",
    "    app = IPython.Application.instance()\n",
    "    app.kernel.do_shutdown(True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "check_versions"
   },
   "source": [
    "Check the versions of the packages you installed.  The KFP SDK version should be >=1.6."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "check_versions:kfp"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "KFP SDK version: 1.8.14\n"
     ]
    }
   ],
   "source": [
    "! python3 -c \"import kfp; print('KFP SDK version: {}'.format(kfp.__version__))\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "set_project_id"
   },
   "outputs": [],
   "source": [
    "PROJECT_ID = \"qwiklabs-gcp-01-d69e76763702\"  # @param {type:\"string\"}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "set_gcloud_project_id"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Updated property [core/project].\n"
     ]
    }
   ],
   "source": [
    "! gcloud config set project $PROJECT_ID"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "region"
   },
   "source": [
    "#### Region\n",
    "\n",
    "You can also change the `REGION` variable, which is used for operations\n",
    "throughout the rest of this notebook.  Below are regions supported for Vertex AI. We recommend that you choose the region closest to you.\n",
    "\n",
    "- Americas: `us-central1`\n",
    "- Europe: `europe-west4`\n",
    "- Asia Pacific: `asia-east1`\n",
    "\n",
    "You may not use a multi-regional bucket for training with Vertex AI. Not all regions provide support for all Vertex AI services.\n",
    "\n",
    "Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "04933ed28eef"
   },
   "outputs": [],
   "source": [
    "REGION = \"us-central1\"  # @param {type:\"string\"}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "id": "jvNx3KyF2Ou0"
   },
   "outputs": [],
   "source": [
    "if REGION == \"[your-region]\":\n",
    "    REGION = \"us-central1\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "timestamp"
   },
   "source": [
    "#### Timestamp\n",
    "\n",
    "If you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, you create a timestamp for each instance session, and append the timestamp onto the name of resources you create in this tutorial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "id": "lFCteCD32Ou0"
   },
   "outputs": [],
   "source": [
    "from datetime import datetime\n",
    "\n",
    "TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "bucket:mbsdk"
   },
   "source": [
    "### Create a Cloud Storage bucket\n",
    "\n",
    "**The following steps are required, regardless of your notebook environment.**\n",
    "\n",
    "When you initialize the Vertex AI SDK for Python, you specify a Cloud Storage staging bucket. The staging bucket is where all the data associated with your dataset and model resources are retained across sessions.\n",
    "\n",
    "Set the name of your Cloud Storage bucket below. Bucket names must be globally unique across all Google Cloud projects, including those outside of your organization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "id": "bucket"
   },
   "outputs": [],
   "source": [
    "BUCKET_NAME = \"[your-bucket-name]\"  # Replace with your bucket name\n",
    "BUCKET_URI = f\"gs://{BUCKET_NAME}\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "autoset_bucket"
   },
   "outputs": [],
   "source": [
    "if BUCKET_NAME == \"\" or BUCKET_NAME is None or BUCKET_NAME == \"[your-bucket-name]\":\n",
    "    BUCKET_NAME = PROJECT_ID + \"aip-\" + TIMESTAMP\n",
    "    BUCKET_URI = \"gs://\" + BUCKET_NAME"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "create_bucket"
   },
   "source": [
    "**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "id": "IXf7nYUc2Ou2"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating gs://qwiklabs-gcp-01-d69e76763702/...\n"
     ]
    }
   ],
   "source": [
    "! gsutil mb -l $REGION $BUCKET_URI"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "validate_bucket"
   },
   "source": [
    "Finally, validate access to your Cloud Storage bucket by examining its contents:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "id": "p23SI9T42Ou2"
   },
   "outputs": [],
   "source": [
    "! gsutil ls -al $BUCKET_URI"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "set_service_account"
   },
   "source": [
    "#### Service Account\n",
    "\n",
    "**If you don't know your service account**, try to get your service account using `gcloud` command by executing the second cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "id": "NY-zkA0Y2Ou3"
   },
   "outputs": [],
   "source": [
    "SERVICE_ACCOUNT = \"[your-service-account]\"  # Replace with your service account"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "set_service_account:pipelines"
   },
   "source": [
    "#### Set service account access for Vertex AI Pipelines\n",
    "\n",
    "Run the following commands to grant your service account access to read and write pipeline artifacts in the bucket that you created in the previous step -- you only need to run these once per service account."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "id": "0QNsyzEF2Ou4"
   },
   "outputs": [],
   "source": [
    "! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectCreator $BUCKET_URI\n",
    "\n",
    "! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectViewer $BUCKET_URI"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "setup_vars"
   },
   "source": [
    "### Set up variables\n",
    "\n",
    "Next, set up some variables used throughout the tutorial.\n",
    "### Import libraries and define constants"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "id": "import_aip:mbsdk"
   },
   "outputs": [],
   "source": [
    "from typing import NamedTuple\n",
    "\n",
    "import google.cloud.aiplatform as aip\n",
    "from kfp import dsl\n",
    "from kfp.v2 import compiler\n",
    "from kfp.v2.dsl import component"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "aip_constants:endpoint"
   },
   "source": [
    "#### Vertex AI constants\n",
    "\n",
    "Setup up the following constants for Vertex AI:\n",
    "\n",
    "- `API_ENDPOINT`: The Vertex AI API service endpoint for `Dataset`, `Model`, `Job`, `Pipeline` and `Endpoint` services."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "id": "TnO8gVBb2Ou4"
   },
   "outputs": [],
   "source": [
    "# API service endpoint\n",
    "API_ENDPOINT = \"{}-aiplatform.googleapis.com\".format(REGION)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "pipeline_constants"
   },
   "source": [
    "#### Vertex AI Pipelines constants\n",
    "\n",
    "Setup up the following constants for Vertex AI Pipelines:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "id": "ONHtQDz32Ou5"
   },
   "outputs": [],
   "source": [
    "PIPELINE_ROOT = \"{}/pipeline_root/intro\".format(BUCKET_URI)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "init_aip:mbsdk"
   },
   "source": [
    "## Initialize Vertex AI SDK for Python\n",
    "\n",
    "Initialize the Vertex AI SDK for Python for your project and corresponding bucket."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "id": "mbpw7oyM2Ou5"
   },
   "outputs": [],
   "source": [
    "# Initialize the Vertex AI SDK for your project and bucket\n",
    "# TODO 1: Your code goes here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "define_component:hello_world"
   },
   "source": [
    "### Define Python function-based pipeline components\n",
    "\n",
    "In this tutorial, you define a simple pipeline that has three steps, where each step is defined as a component.\n",
    "\n",
    "#### Define hello_world component\n",
    "\n",
    "First, define a component based on a very simple Python function. It takes a string input parameter and returns that value as output.\n",
    "\n",
    "Note the use of the `@component` decorator, which compiles the function to a KFP component when evaluated.  For example purposes, this example specifies a base image to use for the component (`python:3.9`), and a component YAML file, `hw.yaml`. The compiled component specification is written to this file.  (The default base image is `python:3.7`, which would of course work just fine too)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "id": "GjJhJUID2Ou6"
   },
   "outputs": [],
   "source": [
    "@component(output_component_file=\"hw.yaml\", base_image=\"python:3.9\")\n",
    "def hello_world(text: str) -> str:\n",
    "    print(text)\n",
    "    return text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "SWcIXuxR2Ou6"
   },
   "source": [
    "As you'll see below, compilation of this component creates a [task factory function](https://www.kubeflow.org/docs/components/pipelines/sdk/python-function-components/)—called `hello_world`— that you can use in defining a pipeline step.\n",
    "\n",
    "While not shown here, if you want to share this component definition, or use it in another context, you could also load it from its yaml file like this:\n",
    "`hello_world_op = components.load_component_from_file('./hw.yaml')`.\n",
    "You can also use the `load_component_from_url` method, if your component yaml file is stored online. (For GitHub URLs, load the 'raw' file.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "define_component:two_outputs"
   },
   "source": [
    "#### Define two_outputs component\n",
    "\n",
    "The first component below, `two_outputs`, demonstrates installing a package -- in this case the `google-cloud-storage` package. Alternatively, you can specify a base image that includes the necessary installations.\n",
    "\n",
    "*Note:* The component function won't actually use the package.\n",
    "\n",
    "Alternatively, you can specify a base image that includes the necessary installations.\n",
    "\n",
    "The `two_outputs` component returns two named outputs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "id": "U4Yv33su2Ou6"
   },
   "outputs": [],
   "source": [
    "@component(packages_to_install=[\"google-cloud-storage\"])\n",
    "def two_outputs(\n",
    "    text: str,\n",
    ") -> NamedTuple(\n",
    "    \"Outputs\",\n",
    "    [\n",
    "        (\"output_one\", str),  # Return parameters\n",
    "        (\"output_two\", str),\n",
    "    ],\n",
    "):\n",
    "    # the import is not actually used for this simple example, but the import\n",
    "    # is successful, as it was included in the `packages_to_install` list.\n",
    "    from google.cloud import storage  # noqa: F401\n",
    "\n",
    "    o1 = f\"output one from text: {text}\"\n",
    "    o2 = f\"output two from text: {text}\"\n",
    "    print(\"output one: {}; output_two: {}\".format(o1, o2))\n",
    "    return (o1, o2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "define_component:consumer"
   },
   "source": [
    "#### Define the consumer component\n",
    "\n",
    "The third component, `consumer`, takes three string inputs and prints them out."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "id": "bu8XvOj82Ou6"
   },
   "outputs": [],
   "source": [
    "@component\n",
    "def consumer(text1: str, text2: str, text3: str):\n",
    "    print(f\"text1: {text1}; text2: {text2}; text3: {text3}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "define_pipeline:intro"
   },
   "source": [
    "### Define a pipeline that uses the components\n",
    "\n",
    "Next, define a pipeline that uses these three components.\n",
    "\n",
    "By evaluating the component definitions above, you've created task factory functions that are used in the pipeline definition to create the pipeline steps.\n",
    "\n",
    "The pipeline takes an input parameter, and passes that parameter as an argument to the first two pipeline steps (`hw_task` and `two_outputs_task`).\n",
    "\n",
    "Then, the third pipeline step (`consumer_task`) consumes the outputs of the first and second steps.  Because the `hello_world` component definition just returns one unnamed output, you refer to it as `hw_task.output`.  The `two_outputs` task returns two named outputs, which you access as `two_outputs_task.outputs[\"<output_name>\"]`.\n",
    "\n",
    "*Note:* In the `@dsl.pipeline` decorator, you're defining the `PIPELINE_ROOT` Cloud Storage path to use.  If you had not included that info here, it would be required to specify it when creating the pipeline run, as you'll see below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "id": "CV5dRAeJ2Ou7"
   },
   "outputs": [],
   "source": [
    "@dsl.pipeline(\n",
    "    name=\"hello-world-v2\",\n",
    "    description=\"A simple intro pipeline\",\n",
    "    pipeline_root=PIPELINE_ROOT,\n",
    ")\n",
    "def pipeline(text: str = \"hi there\"):\n",
    "    hw_task = hello_world(text)\n",
    "    two_outputs_task = two_outputs(text)\n",
    "    consumer_task = consumer(  # noqa: F841\n",
    "        hw_task.output,\n",
    "        two_outputs_task.outputs[\"output_one\"],\n",
    "        two_outputs_task.outputs[\"output_two\"],\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "compile_pipeline"
   },
   "source": [
    "## Compile the pipeline\n",
    "\n",
    "Next, compile the pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "id": "VP_JJ9Oe2Ou7"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/jupyter/.local/lib/python3.7/site-packages/kfp/v2/compiler/compiler.py:1293: FutureWarning: APIs imported from the v1 namespace (e.g. kfp.dsl, kfp.components, etc) will not be supported by the v2 compiler since v2.0.0\n",
      "  category=FutureWarning,\n"
     ]
    }
   ],
   "source": [
    "# Compile the pipeline\n",
    "from kfp.v2 import compiler  # noqa: F811\n",
    "\n",
    "compiler.Compiler().compile(# TODO 2: Your code goes here)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "run_pipeline:intro"
   },
   "source": [
    "## Run the pipeline\n",
    "\n",
    "Next, run the pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "id": "sjxaBix_2Ou7"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating PipelineJob\n",
      "PipelineJob created. Resource name: projects/192756378758/locations/us-central1/pipelineJobs/hello-world-v2-20220922141223\n",
      "To use this PipelineJob in another session:\n",
      "pipeline_job = aiplatform.PipelineJob.get('projects/192756378758/locations/us-central1/pipelineJobs/hello-world-v2-20220922141223')\n",
      "View Pipeline Job:\n",
      "https://console.cloud.google.com/vertex-ai/locations/us-central1/pipelines/runs/hello-world-v2-20220922141223?project=192756378758\n",
      "PipelineJob projects/192756378758/locations/us-central1/pipelineJobs/hello-world-v2-20220922141223 current state:\n",
      "PipelineState.PIPELINE_STATE_PENDING\n",
      "PipelineJob projects/192756378758/locations/us-central1/pipelineJobs/hello-world-v2-20220922141223 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "PipelineJob projects/192756378758/locations/us-central1/pipelineJobs/hello-world-v2-20220922141223 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "PipelineJob projects/192756378758/locations/us-central1/pipelineJobs/hello-world-v2-20220922141223 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "PipelineJob projects/192756378758/locations/us-central1/pipelineJobs/hello-world-v2-20220922141223 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "PipelineJob run completed. Resource name: projects/192756378758/locations/us-central1/pipelineJobs/hello-world-v2-20220922141223\n"
     ]
    }
   ],
   "source": [
    "# Create and run the pipeline job run\n",
    "DISPLAY_NAME = \"intro_\" + TIMESTAMP\n",
    "\n",
    "job = # TODO 3: Your code goes here\n",
    "\n",
    "job.run()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "view_pipeline_run:intro"
   },
   "source": [
    "Click on the generated link to see your run in the Cloud Console.\n",
    "\n",
    "<!-- It should look something like this as it is running:\n",
    "\n",
    "<a href=\"https://storage.googleapis.com/amy-jo/images/mp/automl_tabular_classif.png\" target=\"_blank\"><img src=\"https://storage.googleapis.com/amy-jo/images/mp/automl_tabular_classif.png\" width=\"40%\"/></a> -->\n",
    "\n",
    "In the UI, many of the pipeline DAG nodes will expand or collapse when you click on them. Here is a partially-expanded view of the DAG (click image to see larger version).\n",
    "\n",
    "<a href=\"https://storage.googleapis.com/amy-jo/images/mp/intro_pipeline.png\" target=\"_blank\"><img src=\"https://storage.googleapis.com/amy-jo/images/mp/intro_pipeline.png\" width=\"60%\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "d428cc8803e7"
   },
   "source": [
    "### Delete the pipeline job\n",
    "\n",
    "You can delete the pipeline job with the method `delete()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "id": "97802d9432e7"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Deleting PipelineJob : projects/192756378758/locations/us-central1/pipelineJobs/hello-world-v2-20220922141223\n",
      "Delete PipelineJob  backing LRO: projects/192756378758/locations/us-central1/operations/5955010295213588480\n",
      "PipelineJob deleted. . Resource name: projects/192756378758/locations/us-central1/pipelineJobs/hello-world-v2-20220922141223\n"
     ]
    }
   ],
   "source": [
    "job.delete()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "schedule_pipeline_run"
   },
   "source": [
    "## Recurring pipeline runs: create a scheduled pipeline job\n",
    "\n",
    "This section shows how to create a **scheduled pipeline job**.  You do this using the pipeline you already defined.\n",
    "\n",
    "Under the hood, the scheduled jobs are supported by the Cloud Scheduler and a Cloud Functions function.  Check first that the APIs for both of these services are enabled.\n",
    "**You will need to first enable the [enable the Cloud Scheduler API](http://console.cloud.google.com/apis/library/cloudscheduler.googleapis.com) and the [Cloud Functions and Cloud Build APIs](https://console.cloud.google.com/flows/enableapi?apiid=cloudfunctions,cloudbuild.googleapis.com) if you have not already done so.**\n",
    "**Note:** you need to [create an App Engine app for your project](https://cloud.google.com/scheduler/docs/quickstart) if one does not already exist.\n",
    "\n",
    "\n",
    "See the [Cloud Scheduler](https://cloud.google.com/scheduler/docs/configuring/cron-job-schedules) documentation for more on the cron syntax.\n",
    "\n",
    "Create a scheduled pipeline job, passing as an argument the job specification file that you compiled above.\n",
    "\n",
    "*Note:* You can pass a `parameter_values` dict that specifies the pipeline input parameters you want to use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "id": "Ty5hDoNX2Ou8"
   },
   "outputs": [],
   "source": [
    "if not os.getenv(\"IS_TESTING\"):\n",
    "    from kfp.v2.google.client import AIPlatformClient  # noqa: F811\n",
    "\n",
    "    api_client = AIPlatformClient(project_id=PROJECT_ID, region=REGION)\n",
    "\n",
    "    # adjust time zone and cron schedule as necessary\n",
    "    response = api_client.create_schedule_from_job_spec(\n",
    "        job_spec_path=\"intro_pipeline.json\",\n",
    "        schedule=\"2 * * * *\",\n",
    "        time_zone=\"America/Los_Angeles\",  # change this as necessary\n",
    "        parameter_values={\"text\": \"Hello world!\"},\n",
    "        # pipeline_root=PIPELINE_ROOT  # this argument is necessary if you did not specify PIPELINE_ROOT as part of the pipeline definition.\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "J8AP1viy2Ou8"
   },
   "source": [
    "Once the scheduled job is created, you can see it listed in the [Cloud Scheduler](https://console.cloud.google.com/cloudscheduler/) panel in the Console.\n",
    "\n",
    "<a href=\"https://storage.googleapis.com/amy-jo/images/kf-pls/pipelines_scheduler.png\" target=\"_blank\"><img src=\"https://storage.googleapis.com/amy-jo/images/kf-pls/pipelines_scheduler.png\" width=\"95%\"/></a>\n",
    "\n",
    "You can test the setup from the Cloud Scheduler panel by clicking 'RUN NOW'.\n",
    "\n",
    "> **Note**: The implementation is using a Cloud Functions function, which you can see listed in the [Cloud Functions](https://console.cloud.google.com/functions/list) panel in the console as `templated_http_request-v1`.\n",
    "Don't delete this function, as it will prevent the Cloud Scheduler jobs from actually kicking off the pipeline run.  If you do delete it, create a new scheduled job in order to recreate the function.\n",
    "\n",
    "When you're done experimenting, you probably want to **PAUSE** your scheduled job from the Cloud Scheduler panel, so that the recurrent jobs do not keep running."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "run_pipeline_service_account"
   },
   "source": [
    "## Specifying a service account to use for a pipeline run\n",
    "\n",
    "By default, the [service account](https://cloud.google.com/iam/docs/service-accounts) used for your pipeline run is your [default compute engine service account](https://cloud.google.com/compute/docs/access/service-accounts#default_service_account).\n",
    "However, you might want to run pipelines with permissions to access different roles than those configured for your default SA (e.g. perhaps using a more restricted set of permissions).\n",
    "\n",
    "If you want to execute your pipeline using a different service account, this is straightforward to do.  You just need to give the new service account the correct permissions.\n",
    "\n",
    "### Create a service account\n",
    "\n",
    "Once your service account is configured, you pass it as an argument to the `create_run_from_job_spec` method. The pipeline job runs with the permissions of the given service account.\n",
    "\n",
    "Learn about [creating and configuring a service account to work with Vertex AI Pipelines](https://cloud.google.com/vertex-ai/docs/pipelines/configure-project#service-account)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "id": "CHcFxq742Ou9"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "See the Pipeline job <a href=\"https://console.cloud.google.com/vertex-ai/locations/us-central1/pipelines/runs/hello-world-v2-20220922143209?project=qwiklabs-gcp-01-d69e76763702\" target=\"_blank\" >here</a>."
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "if not os.getenv(\"IS_TESTING\"):\n",
    "    response = api_client.create_run_from_job_spec(\n",
    "        job_spec_path=\"intro_pipeline.json\",\n",
    "        pipeline_root=PIPELINE_ROOT,\n",
    "        service_account=SERVICE_ACCOUNT,  # <-- CHANGE to use non-default service account\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "run_pipeline_caching"
   },
   "source": [
    "## Pipeline step caching\n",
    "\n",
    "By default, pipeline step caching is enabled. This means that the results of previous step executions are reused when possible.\n",
    "\n",
    "if you want to disable caching for a pipeline run, you can pass the `enable_caching=False` argument to the `PipelineJob` constructor when you submit the pipeline job, as shown below.  Try submitting the example pipeline job again, first with and then without caching enabled."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "id": "VrM5wqjh2Ou9"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating PipelineJob\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.pipeline_jobs:Creating PipelineJob\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PipelineJob created. Resource name: projects/192756378758/locations/us-central1/pipelineJobs/hello-world-v2-20220922143250\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.pipeline_jobs:PipelineJob created. Resource name: projects/192756378758/locations/us-central1/pipelineJobs/hello-world-v2-20220922143250\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "To use this PipelineJob in another session:\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.pipeline_jobs:To use this PipelineJob in another session:\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pipeline_job = aiplatform.PipelineJob.get('projects/192756378758/locations/us-central1/pipelineJobs/hello-world-v2-20220922143250')\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.pipeline_jobs:pipeline_job = aiplatform.PipelineJob.get('projects/192756378758/locations/us-central1/pipelineJobs/hello-world-v2-20220922143250')\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "View Pipeline Job:\n",
      "https://console.cloud.google.com/vertex-ai/locations/us-central1/pipelines/runs/hello-world-v2-20220922143250?project=192756378758\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.pipeline_jobs:View Pipeline Job:\n",
      "https://console.cloud.google.com/vertex-ai/locations/us-central1/pipelines/runs/hello-world-v2-20220922143250?project=192756378758\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PipelineJob projects/192756378758/locations/us-central1/pipelineJobs/hello-world-v2-20220922143250 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.pipeline_jobs:PipelineJob projects/192756378758/locations/us-central1/pipelineJobs/hello-world-v2-20220922143250 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PipelineJob projects/192756378758/locations/us-central1/pipelineJobs/hello-world-v2-20220922143250 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.pipeline_jobs:PipelineJob projects/192756378758/locations/us-central1/pipelineJobs/hello-world-v2-20220922143250 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PipelineJob projects/192756378758/locations/us-central1/pipelineJobs/hello-world-v2-20220922143250 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.pipeline_jobs:PipelineJob projects/192756378758/locations/us-central1/pipelineJobs/hello-world-v2-20220922143250 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PipelineJob projects/192756378758/locations/us-central1/pipelineJobs/hello-world-v2-20220922143250 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.pipeline_jobs:PipelineJob projects/192756378758/locations/us-central1/pipelineJobs/hello-world-v2-20220922143250 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PipelineJob projects/192756378758/locations/us-central1/pipelineJobs/hello-world-v2-20220922143250 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.pipeline_jobs:PipelineJob projects/192756378758/locations/us-central1/pipelineJobs/hello-world-v2-20220922143250 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PipelineJob run completed. Resource name: projects/192756378758/locations/us-central1/pipelineJobs/hello-world-v2-20220922143250\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.pipeline_jobs:PipelineJob run completed. Resource name: projects/192756378758/locations/us-central1/pipelineJobs/hello-world-v2-20220922143250\n"
     ]
    }
   ],
   "source": [
    "# Disable caching for a pipeline run\n",
    "job = aip.PipelineJob(# TODO 4: Your code goes here\n",
    ")\n",
    "\n",
    "job.run()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "37e2b84a0c0d"
   },
   "source": [
    "### Delete the pipeline job\n",
    "\n",
    "You can delete the pipeline job with the method `delete()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "id": "c5926e7eaad8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Deleting PipelineJob : projects/192756378758/locations/us-central1/pipelineJobs/hello-world-v2-20220922143250\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.base:Deleting PipelineJob : projects/192756378758/locations/us-central1/pipelineJobs/hello-world-v2-20220922143250\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Delete PipelineJob  backing LRO: projects/192756378758/locations/us-central1/operations/8227076307222003712\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.base:Delete PipelineJob  backing LRO: projects/192756378758/locations/us-central1/operations/8227076307222003712\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PipelineJob deleted. . Resource name: projects/192756378758/locations/us-central1/pipelineJobs/hello-world-v2-20220922143250\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.base:PipelineJob deleted. . Resource name: projects/192756378758/locations/us-central1/pipelineJobs/hello-world-v2-20220922143250\n"
     ]
    }
   ],
   "source": [
    "job.delete()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "pipelines_rest"
   },
   "source": [
    "## Using the Pipelines REST API\n",
    "\n",
    "At times you may want to use the REST API instead of the Python KFP SDK.  Below are examples of how to do that.\n",
    "\n",
    "Where a command requires a pipeline ID, you can get that info from the \"Run\" column in the pipelines list— as shown below— as well as from the 'details' page for a given pipeline.  You can also see that info when you list pipeline job(s) via the API.\n",
    "\n",
    "![pipeline.png]()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "pipelines_rest_list"
   },
   "source": [
    "### List pipeline jobs\n",
    "\n",
    "*Note:* This request may generate a large response if you have many pipeline runs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "id": "1nJSqELi2Ou-"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"pipelineJobs\": [\n",
      "    {\n",
      "      \"name\": \"projects/192756378758/locations/us-central1/pipelineJobs/hello-world-v2-20220922143209\",\n",
      "      \"displayName\": \"hello-world-v2-20220922143209\",\n",
      "      \"createTime\": \"2022-09-22T14:32:09.858920Z\",\n",
      "      \"startTime\": \"2022-09-22T14:32:10.420175Z\",\n",
      "      \"endTime\": \"2022-09-22T14:32:12.894671Z\",\n",
      "      \"updateTime\": \"2022-09-22T14:32:12.894671Z\",\n",
      "      \"pipelineSpec\": {\n",
      "        \"deploymentConfig\": {\n",
      "          \"@type\": \"type.googleapis.com/ml_pipelines.PipelineDeploymentConfig\",\n",
      "          \"executors\": {\n",
      "            \"exec-consumer\": {\n",
      "              \"container\": {\n",
      "                \"image\": \"python:3.7\",\n",
      "                \"command\": [\n",
      "                  \"sh\",\n",
      "                  \"-c\",\n",
      "                  \"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet     --no-warn-script-location 'kfp==1.8.14' && \\\"$0\\\" \\\"$@\\\"\\n\",\n",
      "                  \"sh\",\n",
      "                  \"-ec\",\n",
      "                  \"program_path=$(mktemp -d)\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\npython3 -m kfp.v2.components.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\n",
      "                  \"\\nimport kfp\\nfrom kfp.v2 import dsl\\nfrom kfp.v2.dsl import *\\nfrom typing import *\\n\\ndef consumer(text1: str, text2: str, text3: str):\\n    print(f\\\"text1: {text1}; text2: {text2}; text3: {text3}\\\")\\n\\n\"\n",
      "                ],\n",
      "                \"args\": [\n",
      "                  \"--executor_input\",\n",
      "                  \"{{$}}\",\n",
      "                  \"--function_to_execute\",\n",
      "                  \"consumer\"\n",
      "                ]\n",
      "              }\n",
      "            },\n",
      "            \"exec-hello-world\": {\n",
      "              \"container\": {\n",
      "                \"image\": \"python:3.9\",\n",
      "                \"command\": [\n",
      "                  \"sh\",\n",
      "                  \"-c\",\n",
      "                  \"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet     --no-warn-script-location 'kfp==1.8.14' && \\\"$0\\\" \\\"$@\\\"\\n\",\n",
      "                  \"sh\",\n",
      "                  \"-ec\",\n",
      "                  \"program_path=$(mktemp -d)\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\npython3 -m kfp.v2.components.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\n",
      "                  \"\\nimport kfp\\nfrom kfp.v2 import dsl\\nfrom kfp.v2.dsl import *\\nfrom typing import *\\n\\ndef hello_world(text: str) -\\u003e str:\\n    print(text)\\n    return text\\n\\n\"\n",
      "                ],\n",
      "                \"args\": [\n",
      "                  \"--executor_input\",\n",
      "                  \"{{$}}\",\n",
      "                  \"--function_to_execute\",\n",
      "                  \"hello_world\"\n",
      "                ]\n",
      "              }\n",
      "            },\n",
      "            \"exec-two-outputs\": {\n",
      "              \"container\": {\n",
      "                \"image\": \"python:3.7\",\n",
      "                \"command\": [\n",
      "                  \"sh\",\n",
      "                  \"-c\",\n",
      "                  \"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet     --no-warn-script-location 'google-cloud-storage' 'kfp==1.8.14' && \\\"$0\\\" \\\"$@\\\"\\n\",\n",
      "                  \"sh\",\n",
      "                  \"-ec\",\n",
      "                  \"program_path=$(mktemp -d)\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\npython3 -m kfp.v2.components.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\n",
      "                  \"\\nimport kfp\\nfrom kfp.v2 import dsl\\nfrom kfp.v2.dsl import *\\nfrom typing import *\\n\\ndef two_outputs(\\n    text: str,\\n) -\\u003e NamedTuple(\\n    \\\"Outputs\\\",\\n    [\\n        (\\\"output_one\\\", str),  # Return parameters\\n        (\\\"output_two\\\", str),\\n    ],\\n):\\n    # the import is not actually used for this simple example, but the import\\n    # is successful, as it was included in the `packages_to_install` list.\\n    from google.cloud import storage  # noqa: F401\\n\\n    o1 = f\\\"output one from text: {text}\\\"\\n    o2 = f\\\"output two from text: {text}\\\"\\n    print(\\\"output one: {}; output_two: {}\\\".format(o1, o2))\\n    return (o1, o2)\\n\\n\"\n",
      "                ],\n",
      "                \"args\": [\n",
      "                  \"--executor_input\",\n",
      "                  \"{{$}}\",\n",
      "                  \"--function_to_execute\",\n",
      "                  \"two_outputs\"\n",
      "                ]\n",
      "              }\n",
      "            }\n",
      "          }\n",
      "        },\n",
      "        \"components\": {\n",
      "          \"comp-consumer\": {\n",
      "            \"inputDefinitions\": {\n",
      "              \"parameters\": {\n",
      "                \"text1\": {\n",
      "                  \"type\": \"STRING\"\n",
      "                },\n",
      "                \"text2\": {\n",
      "                  \"type\": \"STRING\"\n",
      "                },\n",
      "                \"text3\": {\n",
      "                  \"type\": \"STRING\"\n",
      "                }\n",
      "              }\n",
      "            },\n",
      "            \"executorLabel\": \"exec-consumer\"\n",
      "          },\n",
      "          \"comp-hello-world\": {\n",
      "            \"inputDefinitions\": {\n",
      "              \"parameters\": {\n",
      "                \"text\": {\n",
      "                  \"type\": \"STRING\"\n",
      "                }\n",
      "              }\n",
      "            },\n",
      "            \"outputDefinitions\": {\n",
      "              \"parameters\": {\n",
      "                \"Output\": {\n",
      "                  \"type\": \"STRING\"\n",
      "                }\n",
      "              }\n",
      "            },\n",
      "            \"executorLabel\": \"exec-hello-world\"\n",
      "          },\n",
      "          \"comp-two-outputs\": {\n",
      "            \"inputDefinitions\": {\n",
      "              \"parameters\": {\n",
      "                \"text\": {\n",
      "                  \"type\": \"STRING\"\n",
      "                }\n",
      "              }\n",
      "            },\n",
      "            \"outputDefinitions\": {\n",
      "              \"parameters\": {\n",
      "                \"output_one\": {\n",
      "                  \"type\": \"STRING\"\n",
      "                },\n",
      "                \"output_two\": {\n",
      "                  \"type\": \"STRING\"\n",
      "                }\n",
      "              }\n",
      "            },\n",
      "            \"executorLabel\": \"exec-two-outputs\"\n",
      "          }\n",
      "        },\n",
      "        \"schemaVersion\": \"2.0.0\",\n",
      "        \"root\": {\n",
      "          \"inputDefinitions\": {\n",
      "            \"parameters\": {\n",
      "              \"text\": {\n",
      "                \"type\": \"STRING\"\n",
      "              }\n",
      "            }\n",
      "          },\n",
      "          \"dag\": {\n",
      "            \"tasks\": {\n",
      "              \"consumer\": {\n",
      "                \"taskInfo\": {\n",
      "                  \"name\": \"consumer\"\n",
      "                },\n",
      "                \"inputs\": {\n",
      "                  \"parameters\": {\n",
      "                    \"text1\": {\n",
      "                      \"taskOutputParameter\": {\n",
      "                        \"producerTask\": \"hello-world\",\n",
      "                        \"outputParameterKey\": \"Output\"\n",
      "                      }\n",
      "                    },\n",
      "                    \"text2\": {\n",
      "                      \"taskOutputParameter\": {\n",
      "                        \"producerTask\": \"two-outputs\",\n",
      "                        \"outputParameterKey\": \"output_one\"\n",
      "                      }\n",
      "                    },\n",
      "                    \"text3\": {\n",
      "                      \"taskOutputParameter\": {\n",
      "                        \"producerTask\": \"two-outputs\",\n",
      "                        \"outputParameterKey\": \"output_two\"\n",
      "                      }\n",
      "                    }\n",
      "                  }\n",
      "                },\n",
      "                \"dependentTasks\": [\n",
      "                  \"hello-world\",\n",
      "                  \"two-outputs\"\n",
      "                ],\n",
      "                \"cachingOptions\": {\n",
      "                  \"enableCache\": true\n",
      "                },\n",
      "                \"componentRef\": {\n",
      "                  \"name\": \"comp-consumer\"\n",
      "                }\n",
      "              },\n",
      "              \"hello-world\": {\n",
      "                \"taskInfo\": {\n",
      "                  \"name\": \"hello-world\"\n",
      "                },\n",
      "                \"inputs\": {\n",
      "                  \"parameters\": {\n",
      "                    \"text\": {\n",
      "                      \"componentInputParameter\": \"text\"\n",
      "                    }\n",
      "                  }\n",
      "                },\n",
      "                \"cachingOptions\": {\n",
      "                  \"enableCache\": true\n",
      "                },\n",
      "                \"componentRef\": {\n",
      "                  \"name\": \"comp-hello-world\"\n",
      "                }\n",
      "              },\n",
      "              \"two-outputs\": {\n",
      "                \"taskInfo\": {\n",
      "                  \"name\": \"two-outputs\"\n",
      "                },\n",
      "                \"inputs\": {\n",
      "                  \"parameters\": {\n",
      "                    \"text\": {\n",
      "                      \"componentInputParameter\": \"text\"\n",
      "                    }\n",
      "                  }\n",
      "                },\n",
      "                \"cachingOptions\": {\n",
      "                  \"enableCache\": true\n",
      "                },\n",
      "                \"componentRef\": {\n",
      "                  \"name\": \"comp-two-outputs\"\n",
      "                }\n",
      "              }\n",
      "            }\n",
      "          }\n",
      "        },\n",
      "        \"sdkVersion\": \"kfp-1.8.14\",\n",
      "        \"pipelineInfo\": {\n",
      "          \"name\": \"hello-world-v2\"\n",
      "        },\n",
      "        \"deploymentSpec\": {\n",
      "          \"executors\": {\n",
      "            \"exec-consumer\": {\n",
      "              \"container\": {\n",
      "                \"image\": \"python:3.7\",\n",
      "                \"command\": [\n",
      "                  \"sh\",\n",
      "                  \"-c\",\n",
      "                  \"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet     --no-warn-script-location 'kfp==1.8.14' && \\\"$0\\\" \\\"$@\\\"\\n\",\n",
      "                  \"sh\",\n",
      "                  \"-ec\",\n",
      "                  \"program_path=$(mktemp -d)\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\npython3 -m kfp.v2.components.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\n",
      "                  \"\\nimport kfp\\nfrom kfp.v2 import dsl\\nfrom kfp.v2.dsl import *\\nfrom typing import *\\n\\ndef consumer(text1: str, text2: str, text3: str):\\n    print(f\\\"text1: {text1}; text2: {text2}; text3: {text3}\\\")\\n\\n\"\n",
      "                ],\n",
      "                \"args\": [\n",
      "                  \"--executor_input\",\n",
      "                  \"{{$}}\",\n",
      "                  \"--function_to_execute\",\n",
      "                  \"consumer\"\n",
      "                ]\n",
      "              }\n",
      "            },\n",
      "            \"exec-hello-world\": {\n",
      "              \"container\": {\n",
      "                \"image\": \"python:3.9\",\n",
      "                \"command\": [\n",
      "                  \"sh\",\n",
      "                  \"-c\",\n",
      "                  \"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet     --no-warn-script-location 'kfp==1.8.14' && \\\"$0\\\" \\\"$@\\\"\\n\",\n",
      "                  \"sh\",\n",
      "                  \"-ec\",\n",
      "                  \"program_path=$(mktemp -d)\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\npython3 -m kfp.v2.components.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\n",
      "                  \"\\nimport kfp\\nfrom kfp.v2 import dsl\\nfrom kfp.v2.dsl import *\\nfrom typing import *\\n\\ndef hello_world(text: str) -\\u003e str:\\n    print(text)\\n    return text\\n\\n\"\n",
      "                ],\n",
      "                \"args\": [\n",
      "                  \"--executor_input\",\n",
      "                  \"{{$}}\",\n",
      "                  \"--function_to_execute\",\n",
      "                  \"hello_world\"\n",
      "                ]\n",
      "              }\n",
      "            },\n",
      "            \"exec-two-outputs\": {\n",
      "              \"container\": {\n",
      "                \"image\": \"python:3.7\",\n",
      "                \"command\": [\n",
      "                  \"sh\",\n",
      "                  \"-c\",\n",
      "                  \"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet     --no-warn-script-location 'google-cloud-storage' 'kfp==1.8.14' && \\\"$0\\\" \\\"$@\\\"\\n\",\n",
      "                  \"sh\",\n",
      "                  \"-ec\",\n",
      "                  \"program_path=$(mktemp -d)\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\npython3 -m kfp.v2.components.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\n",
      "                  \"\\nimport kfp\\nfrom kfp.v2 import dsl\\nfrom kfp.v2.dsl import *\\nfrom typing import *\\n\\ndef two_outputs(\\n    text: str,\\n) -\\u003e NamedTuple(\\n    \\\"Outputs\\\",\\n    [\\n        (\\\"output_one\\\", str),  # Return parameters\\n        (\\\"output_two\\\", str),\\n    ],\\n):\\n    # the import is not actually used for this simple example, but the import\\n    # is successful, as it was included in the `packages_to_install` list.\\n    from google.cloud import storage  # noqa: F401\\n\\n    o1 = f\\\"output one from text: {text}\\\"\\n    o2 = f\\\"output two from text: {text}\\\"\\n    print(\\\"output one: {}; output_two: {}\\\".format(o1, o2))\\n    return (o1, o2)\\n\\n\"\n",
      "                ],\n",
      "                \"args\": [\n",
      "                  \"--executor_input\",\n",
      "                  \"{{$}}\",\n",
      "                  \"--function_to_execute\",\n",
      "                  \"two_outputs\"\n",
      "                ]\n",
      "              }\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "      },\n",
      "      \"state\": \"PIPELINE_STATE_SUCCEEDED\",\n",
      "      \"jobDetail\": {\n",
      "        \"pipelineContext\": {\n",
      "          \"name\": \"projects/192756378758/locations/us-central1/metadataStores/default/contexts/hello-world-v2\"\n",
      "        },\n",
      "        \"pipelineRunContext\": {\n",
      "          \"name\": \"projects/192756378758/locations/us-central1/metadataStores/default/contexts/hello-world-v2-20220922143209\"\n",
      "        }\n",
      "      },\n",
      "      \"runtimeConfig\": {\n",
      "        \"parameters\": {\n",
      "          \"text\": {\n",
      "            \"stringValue\": \"hi there\"\n",
      "          }\n",
      "        },\n",
      "        \"gcsOutputDirectory\": \"gs://qwiklabs-gcp-01-d69e76763702/pipeline_root/intro\"\n",
      "      },\n",
      "      \"serviceAccount\": \"192756378758-compute@developer.gserviceaccount.com\"\n",
      "    }\n",
      "  ]\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "! curl -X GET -H \"Authorization: Bearer $(gcloud auth print-access-token)\"   -H \"Content-Type: application/json\"   https://{API_ENDPOINT}/v1beta1/projects/{PROJECT_ID}/locations/{REGION}/pipelineJobs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "pipelines_rest_create"
   },
   "source": [
    "### Create a pipeline job\n",
    "\n",
    "For this REST request, you submit a compiled pipeline job specification.  We're using the one you generated previously."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "id": "iEUcY8aQ2Ou-"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current', '                                 Dload  Upload   Total   Spent    Left  Speed', '', '  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0{', '  \"name\": \"projects/192756378758/locations/us-central1/pipelineJobs/184807013908217856\",', '  \"createTime\": \"2022-09-22T14:39:04.069052Z\",', '  \"updateTime\": \"2022-09-22T14:39:04.069052Z\",', '  \"pipelineSpec\": {', '    \"deploymentConfig\": {', '      \"@type\": \"type.googleapis.com/ml_pipelines.PipelineDeploymentConfig\",', '      \"executors\": {', '        \"exec-consumer\": {', '          \"container\": {', '            \"image\": \"python:3.7\",', '            \"command\": [', '              \"sh\",', '              \"-c\",', '              \"\\\\nif ! [ -x \\\\\"$(command -v pip)\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet     --no-warn-script-location \\'kfp==1.8.14\\' && \\\\\"$0\\\\\" \\\\\"$@\\\\\"\\\\n\",', '              \"sh\",', '              \"-ec\",', '              \"program_path=$(mktemp -d)\\\\nprintf \\\\\"%s\\\\\" \\\\\"$0\\\\\" \\\\u003e \\\\\"$program_path/ephemeral_component.py\\\\\"\\\\npython3 -m kfp.v2.components.executor_main                         --component_module_path                         \\\\\"$program_path/ephemeral_component.py\\\\\"                         \\\\\"$@\\\\\"\\\\n\",', '              \"\\\\nimport kfp\\\\nfrom kfp.v2 import dsl\\\\nfrom kfp.v2.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef consumer(text1: str, text2: str, text3: str):\\\\n    print(f\\\\\"text1: {text1}; text2: {text2}; text3: {text3}\\\\\")\\\\n\\\\n\"', '            ],', '            \"args\": [', '              \"--executor_input\",', '              \"{{$}}\",', '              \"--function_to_execute\",', '              \"consumer\"', '            ]', '          }', '        },', '        \"exec-hello-world\": {', '          \"container\": {', '            \"image\": \"python:3.9\",', '            \"command\": [', '              \"sh\",', '              \"-c\",', '              \"\\\\nif ! [ -x \\\\\"$(command -v pip)\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet     --no-warn-script-location \\'kfp==1.8.14\\' && \\\\\"$0\\\\\" \\\\\"$@\\\\\"\\\\n\",', '              \"sh\",', '              \"-ec\",', '              \"program_path=$(mktemp -d)\\\\nprintf \\\\\"%s\\\\\" \\\\\"$0\\\\\" \\\\u003e \\\\\"$program_path/ephemeral_component.py\\\\\"\\\\npython3 -m kfp.v2.components.executor_main                         --component_module_path                         \\\\\"$program_path/ephemeral_component.py\\\\\"                         \\\\\"$@\\\\\"\\\\n\",', '              \"\\\\nimport kfp\\\\nfrom kfp.v2 import dsl\\\\nfrom kfp.v2.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef hello_world(text: str) -\\\\u003e str:\\\\n    print(text)\\\\n    return text\\\\n\\\\n\"', '            ],', '            \"args\": [', '              \"--executor_input\",', '              \"{{$}}\",', '              \"--function_to_execute\",', '              \"hello_world\"', '            ]', '          }', '        },', '        \"exec-two-outputs\": {', '          \"container\": {', '            \"image\": \"python:3.7\",', '            \"command\": [', '              \"sh\",', '              \"-c\",', '              \"\\\\nif ! [ -x \\\\\"$(command -v pip)\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet     --no-warn-script-location \\'google-cloud-storage\\' \\'kfp==1.8.14\\' && \\\\\"$0\\\\\" \\\\\"$@\\\\\"\\\\n\",', '              \"sh\",', '              \"-ec\",', '              \"program_path=$(mktemp -d)\\\\nprintf \\\\\"%s\\\\\" \\\\\"$0\\\\\" \\\\u003e \\\\\"$program_path/ephemeral_component.py\\\\\"\\\\npython3 -m kfp.v2.components.executor_main                         --component_module_path                         \\\\\"$program_path/ephemeral_component.py\\\\\"                         \\\\\"$@\\\\\"\\\\n\",', '              \"\\\\nimport kfp\\\\nfrom kfp.v2 import dsl\\\\nfrom kfp.v2.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef two_outputs(\\\\n    text: str,\\\\n) -\\\\u003e NamedTuple(\\\\n    \\\\\"Outputs\\\\\",\\\\n    [\\\\n        (\\\\\"output_one\\\\\", str),  # Return parameters\\\\n        (\\\\\"output_two\\\\\", str),\\\\n    ],\\\\n):\\\\n    # the import is not actually used for this simple example, but the import\\\\n    # is successful, as it was included in the `packages_to_install` list.\\\\n    from google.cloud import storage  # noqa: F401\\\\n\\\\n    o1 = f\\\\\"output one from text: {text}\\\\\"\\\\n    o2 = f\\\\\"output two from text: {text}\\\\\"\\\\n    print(\\\\\"output one: {}; output_two: {}\\\\\".format(o1, o2))\\\\n    return (o1, o2)\\\\n\\\\n\"', '            ],', '            \"args\": [', '              \"--executor_input\",', '              \"{{$}}\",', '              \"--function_to_execute\",', '              \"two_outputs\"', '            ]', '          }', '        }', '      }', '    },', '    \"components\": {', '      \"comp-consumer\": {', '        \"inputDefinitions\": {', '          \"parameters\": {', '            \"text1\": {', '              \"type\": \"STRING\"', '            },', '            \"text2\": {', '              \"type\": \"STRING\"', '            },', '            \"text3\": {', '              \"type\": \"STRING\"', '            }', '          }', '        },', '        \"executorLabel\": \"exec-consumer\"', '      },', '      \"comp-hello-world\": {', '        \"inputDefinitions\": {', '          \"parameters\": {', '            \"text\": {', '              \"type\": \"STRING\"', '            }', '          }', '        },', '        \"outputDefinitions\": {', '          \"parameters\": {', '            \"Output\": {', '              \"type\": \"STRING\"', '            }', '          }', '        },', '        \"executorLabel\": \"exec-hello-world\"', '      },', '      \"comp-two-outputs\": {', '        \"inputDefinitions\": {', '          \"parameters\": {', '            \"text\": {', '              \"type\": \"STRING\"', '            }', '          }', '        },', '        \"outputDefinitions\": {', '          \"parameters\": {', '            \"output_one\": {', '              \"type\": \"STRING\"', '            },', '            \"output_two\": {', '              \"type\": \"STRING\"', '            }', '          }', '        },', '        \"executorLabel\": \"exec-two-outputs\"', '      }', '    },', '    \"schemaVersion\": \"2.0.0\",', '    \"root\": {', '      \"inputDefinitions\": {', '        \"parameters\": {', '          \"text\": {', '            \"type\": \"STRING\"', '          }', '        }', '      },', '      \"dag\": {', '        \"tasks\": {', '          \"consumer\": {', '            \"taskInfo\": {', '              \"name\": \"consumer\"', '            },', '            \"inputs\": {', '              \"parameters\": {', '                \"text1\": {', '                  \"taskOutputParameter\": {', '                    \"producerTask\": \"hello-world\",', '                    \"outputParameterKey\": \"Output\"', '                  }', '                },', '                \"text2\": {', '                  \"taskOutputParameter\": {', '                    \"producerTask\": \"two-outputs\",', '                    \"outputParameterKey\": \"output_one\"', '                  }', '                },', '                \"text3\": {', '                  \"taskOutputParameter\": {', '                    \"producerTask\": \"two-outputs\",', '                    \"outputParameterKey\": \"output_two\"', '                  }', '                }', '              }', '            },', '            \"dependentTasks\": [', '              \"hello-world\",', '              \"two-outputs\"', '            ],', '            \"cachingOptions\": {', '              \"enableCache\": true', '            },', '            \"componentRef\": {', '              \"name\": \"comp-consumer\"', '            }', '          },', '          \"hello-world\": {', '            \"taskInfo\": {', '              \"name\": \"hello-world\"', '            },', '            \"inputs\": {', '              \"parameters\": {', '                \"text\": {', '                  \"componentInputParameter\": \"text\"', '                }', '              }', '            },', '            \"cachingOptions\": {', '              \"enableCache\": true', '            },', '            \"componentRef\": {', '              \"name\": \"comp-hello-world\"', '            }', '          },', '          \"two-outputs\": {', '            \"taskInfo\": {', '              \"name\": \"two-outputs\"', '            },', '            \"inputs\": {', '              \"parameters\": {', '                \"text\": {', '                  \"componentInputParameter\": \"text\"', '                }', '              }', '            },', '            \"cachingOptions\": {', '              \"enableCache\": true', '            },', '            \"componentRef\": {', '              \"name\": \"comp-two-outputs\"', '            }', '          }', '        }', '      }', '    },', '    \"sdkVersion\": \"kfp-1.8.14\",', '    \"pipelineInfo\": {', '      \"name\": \"hello-world-v2\"', '    },', '    \"deploymentSpec\": {', '      \"executors\": {', '        \"exec-consumer\": {', '          \"container\": {', '            \"image\": \"python:3.7\",', '            \"command\": [', '              \"sh\",', '              \"-c\",', '              \"\\\\nif ! [ -x \\\\\"$(command -v pip)\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet     --no-warn-script-location \\'kfp==1.8.14\\' && \\\\\"$0\\\\\" \\\\\"$@\\\\\"\\\\n\",', '              \"sh\",', '              \"-ec\",', '              \"program_path=$(mktemp -d)\\\\nprintf \\\\\"%s\\\\\" \\\\\"$0\\\\\" \\\\u003e \\\\\"$program_path/ephemeral_component.py\\\\\"\\\\npython3 -m kfp.v2.components.executor_main                         --component_module_path                         \\\\\"$program_path/ephemeral_component.py\\\\\"                         \\\\\"$@\\\\\"\\\\n\",', '              \"\\\\nimport kfp\\\\nfrom kfp.v2 import dsl\\\\nfrom kfp.v2.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef consumer(text1: str, text2: str, text3: str):\\\\n    print(f\\\\\"text1: {text1}; text2: {text2}; text3: {text3}\\\\\")\\\\n\\\\n\"', '            ],', '            \"args\": [', '              \"--executor_input\",', '              \"{{$}}\",', '              \"--function_to_execute\",', '              \"consumer\"', '            ]', '          }', '        },', '        \"exec-hello-world\": {', '          \"container\": {', '            \"image\": \"python:3.9\",', '            \"command\": [', '              \"sh\",', '              \"-c\",', '              \"\\\\nif ! [ -x \\\\\"$(command -v pip)\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet     --no-warn-script-location \\'kfp==1.8.14\\' && \\\\\"$0\\\\\" \\\\\"$@\\\\\"\\\\n\",', '              \"sh\",', '              \"-ec\",', '              \"program_path=$(mktemp -d)\\\\nprintf \\\\\"%s\\\\\" \\\\\"$0\\\\\" \\\\u003e \\\\\"$program_path/ephemeral_component.py\\\\\"\\\\npython3 -m kfp.v2.components.executor_main                         --component_module_path                         \\\\\"$program_path/ephemeral_component.py\\\\\"                         \\\\\"$@\\\\\"\\\\n\",', '              \"\\\\nimport kfp\\\\nfrom kfp.v2 import dsl\\\\nfrom kfp.v2.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef hello_world(text: str) -\\\\u003e str:\\\\n    print(text)\\\\n    return text\\\\n\\\\n\"', '            ],', '            \"args\": [', '              \"--executor_input\",', '              \"{{$}}\",', '              \"--function_to_execute\",', '              \"hello_world\"', '            ]', '          }', '        },', '        \"exec-two-outputs\": {', '          \"container\": {', '            \"image\": \"python:3.7\",', '            \"command\": [', '              \"sh\",', '              \"-c\",', '              \"\\\\nif ! [ -x \\\\\"$(command -v pip)\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet     --no-warn-script-location \\'google-cloud-storage\\' \\'kfp==1.8.14\\' && \\\\\"$0\\\\\" \\\\\"$@\\\\\"\\\\n\",', '              \"sh\",', '              \"-ec\",', '              \"program_path=$(mktemp -d)\\\\nprintf \\\\\"%s\\\\\" \\\\\"$0\\\\\" \\\\u003e \\\\\"$program_path/ephemeral_component.py\\\\\"\\\\npython3 -m kfp.v2.components.executor_main                         --component_module_path                         \\\\\"$program_path/ephemeral_component.py\\\\\"                         \\\\\"$@\\\\\"\\\\n\",', '              \"\\\\nimport kfp\\\\nfrom kfp.v2 import dsl\\\\nfrom kfp.v2.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef two_outputs(\\\\n    text: str,\\\\n) -\\\\u003e NamedTuple(\\\\n    \\\\\"Outputs\\\\\",\\\\n    [\\\\n        (\\\\\"output_one\\\\\", str),  # Return parameters\\\\n        (\\\\\"output_two\\\\\", str),\\\\n    ],\\\\n):\\\\n    # the import is not actually used for this simple example, but the import\\\\n    # is successful, as it was included in the `packages_to_install` list.\\\\n    from google.cloud import storage  # noqa: F401\\\\n\\\\n    o1 = f\\\\\"output one from text: {text}\\\\\"\\\\n    o2 = f\\\\\"output two from text: {text}\\\\\"\\\\n    print(\\\\\"output one: {}; output_two: {}\\\\\".format(o1, o2))\\\\n    return (o1, o2)\\\\n\\\\n\"', '            ],', '            \"args\": [', '              \"--executor_input\",', '              \"{{$}}\",', '              \"--function_to_execute\",', '              \"two_outputs\"', '            ]', '          }', '        }', '      }', '    }', '  },', '  \"state\": \"PIPELINE_STATE_PENDING\",', '  \"runtimeConfig\": {', '    \"parameters\": {', '      \"text\": {', '        \"stringValue\": \"hi there\"', '      }', '    },', '    \"gcsOutputDirectory\": \"gs://qwiklabs-gcp-01-d69e76763702/pipeline_root/intro\"', '  },', '  \"serviceAccount\": \"192756378758-compute@de', '100 19942    0 12319  100  7623  33384  20658 --:--:-- --:--:-- --:--:-- 54043', 'veloper.gserviceaccount.com\"', '}']\n"
     ]
    }
   ],
   "source": [
    "output = ! curl -X POST  -H \"Authorization: Bearer $(gcloud auth print-access-token)\" -H \"Content-Type: application/json; charset=utf-8\"   https://{API_ENDPOINT}/v1beta1/projects/{PROJECT_ID}/locations/{REGION}/pipelineJobs  --data \"@intro_pipeline.json\"\n",
    "\n",
    "# Enter the pipeline run id\n",
    "PIPELINE_RUN_ID = # TODO 5: Your code goes here\n",
    "print(output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "pipelines_rest_get"
   },
   "source": [
    "### Get a pipeline job from its ID"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "id": "D-QEplm82Ou-"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"pipelineJobs\": [\n",
      "    {\n",
      "      \"name\": \"projects/192756378758/locations/us-central1/pipelineJobs/184807013908217856\",\n",
      "      \"createTime\": \"2022-09-22T14:39:04.069052Z\",\n",
      "      \"startTime\": \"2022-09-22T14:39:04.631583Z\",\n",
      "      \"endTime\": \"2022-09-22T14:39:07.054511Z\",\n",
      "      \"updateTime\": \"2022-09-22T14:39:07.054511Z\",\n",
      "      \"pipelineSpec\": {\n",
      "        \"deploymentConfig\": {\n",
      "          \"@type\": \"type.googleapis.com/ml_pipelines.PipelineDeploymentConfig\",\n",
      "          \"executors\": {\n",
      "            \"exec-consumer\": {\n",
      "              \"container\": {\n",
      "                \"image\": \"python:3.7\",\n",
      "                \"command\": [\n",
      "                  \"sh\",\n",
      "                  \"-c\",\n",
      "                  \"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet     --no-warn-script-location 'kfp==1.8.14' && \\\"$0\\\" \\\"$@\\\"\\n\",\n",
      "                  \"sh\",\n",
      "                  \"-ec\",\n",
      "                  \"program_path=$(mktemp -d)\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\npython3 -m kfp.v2.components.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\n",
      "                  \"\\nimport kfp\\nfrom kfp.v2 import dsl\\nfrom kfp.v2.dsl import *\\nfrom typing import *\\n\\ndef consumer(text1: str, text2: str, text3: str):\\n    print(f\\\"text1: {text1}; text2: {text2}; text3: {text3}\\\")\\n\\n\"\n",
      "                ],\n",
      "                \"args\": [\n",
      "                  \"--executor_input\",\n",
      "                  \"{{$}}\",\n",
      "                  \"--function_to_execute\",\n",
      "                  \"consumer\"\n",
      "                ]\n",
      "              }\n",
      "            },\n",
      "            \"exec-hello-world\": {\n",
      "              \"container\": {\n",
      "                \"image\": \"python:3.9\",\n",
      "                \"command\": [\n",
      "                  \"sh\",\n",
      "                  \"-c\",\n",
      "                  \"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet     --no-warn-script-location 'kfp==1.8.14' && \\\"$0\\\" \\\"$@\\\"\\n\",\n",
      "                  \"sh\",\n",
      "                  \"-ec\",\n",
      "                  \"program_path=$(mktemp -d)\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\npython3 -m kfp.v2.components.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\n",
      "                  \"\\nimport kfp\\nfrom kfp.v2 import dsl\\nfrom kfp.v2.dsl import *\\nfrom typing import *\\n\\ndef hello_world(text: str) -\\u003e str:\\n    print(text)\\n    return text\\n\\n\"\n",
      "                ],\n",
      "                \"args\": [\n",
      "                  \"--executor_input\",\n",
      "                  \"{{$}}\",\n",
      "                  \"--function_to_execute\",\n",
      "                  \"hello_world\"\n",
      "                ]\n",
      "              }\n",
      "            },\n",
      "            \"exec-two-outputs\": {\n",
      "              \"container\": {\n",
      "                \"image\": \"python:3.7\",\n",
      "                \"command\": [\n",
      "                  \"sh\",\n",
      "                  \"-c\",\n",
      "                  \"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet     --no-warn-script-location 'google-cloud-storage' 'kfp==1.8.14' && \\\"$0\\\" \\\"$@\\\"\\n\",\n",
      "                  \"sh\",\n",
      "                  \"-ec\",\n",
      "                  \"program_path=$(mktemp -d)\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\npython3 -m kfp.v2.components.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\n",
      "                  \"\\nimport kfp\\nfrom kfp.v2 import dsl\\nfrom kfp.v2.dsl import *\\nfrom typing import *\\n\\ndef two_outputs(\\n    text: str,\\n) -\\u003e NamedTuple(\\n    \\\"Outputs\\\",\\n    [\\n        (\\\"output_one\\\", str),  # Return parameters\\n        (\\\"output_two\\\", str),\\n    ],\\n):\\n    # the import is not actually used for this simple example, but the import\\n    # is successful, as it was included in the `packages_to_install` list.\\n    from google.cloud import storage  # noqa: F401\\n\\n    o1 = f\\\"output one from text: {text}\\\"\\n    o2 = f\\\"output two from text: {text}\\\"\\n    print(\\\"output one: {}; output_two: {}\\\".format(o1, o2))\\n    return (o1, o2)\\n\\n\"\n",
      "                ],\n",
      "                \"args\": [\n",
      "                  \"--executor_input\",\n",
      "                  \"{{$}}\",\n",
      "                  \"--function_to_execute\",\n",
      "                  \"two_outputs\"\n",
      "                ]\n",
      "              }\n",
      "            }\n",
      "          }\n",
      "        },\n",
      "        \"components\": {\n",
      "          \"comp-consumer\": {\n",
      "            \"inputDefinitions\": {\n",
      "              \"parameters\": {\n",
      "                \"text1\": {\n",
      "                  \"type\": \"STRING\"\n",
      "                },\n",
      "                \"text2\": {\n",
      "                  \"type\": \"STRING\"\n",
      "                },\n",
      "                \"text3\": {\n",
      "                  \"type\": \"STRING\"\n",
      "                }\n",
      "              }\n",
      "            },\n",
      "            \"executorLabel\": \"exec-consumer\"\n",
      "          },\n",
      "          \"comp-hello-world\": {\n",
      "            \"inputDefinitions\": {\n",
      "              \"parameters\": {\n",
      "                \"text\": {\n",
      "                  \"type\": \"STRING\"\n",
      "                }\n",
      "              }\n",
      "            },\n",
      "            \"outputDefinitions\": {\n",
      "              \"parameters\": {\n",
      "                \"Output\": {\n",
      "                  \"type\": \"STRING\"\n",
      "                }\n",
      "              }\n",
      "            },\n",
      "            \"executorLabel\": \"exec-hello-world\"\n",
      "          },\n",
      "          \"comp-two-outputs\": {\n",
      "            \"inputDefinitions\": {\n",
      "              \"parameters\": {\n",
      "                \"text\": {\n",
      "                  \"type\": \"STRING\"\n",
      "                }\n",
      "              }\n",
      "            },\n",
      "            \"outputDefinitions\": {\n",
      "              \"parameters\": {\n",
      "                \"output_one\": {\n",
      "                  \"type\": \"STRING\"\n",
      "                },\n",
      "                \"output_two\": {\n",
      "                  \"type\": \"STRING\"\n",
      "                }\n",
      "              }\n",
      "            },\n",
      "            \"executorLabel\": \"exec-two-outputs\"\n",
      "          }\n",
      "        },\n",
      "        \"schemaVersion\": \"2.0.0\",\n",
      "        \"root\": {\n",
      "          \"inputDefinitions\": {\n",
      "            \"parameters\": {\n",
      "              \"text\": {\n",
      "                \"type\": \"STRING\"\n",
      "              }\n",
      "            }\n",
      "          },\n",
      "          \"dag\": {\n",
      "            \"tasks\": {\n",
      "              \"consumer\": {\n",
      "                \"taskInfo\": {\n",
      "                  \"name\": \"consumer\"\n",
      "                },\n",
      "                \"inputs\": {\n",
      "                  \"parameters\": {\n",
      "                    \"text1\": {\n",
      "                      \"taskOutputParameter\": {\n",
      "                        \"producerTask\": \"hello-world\",\n",
      "                        \"outputParameterKey\": \"Output\"\n",
      "                      }\n",
      "                    },\n",
      "                    \"text2\": {\n",
      "                      \"taskOutputParameter\": {\n",
      "                        \"producerTask\": \"two-outputs\",\n",
      "                        \"outputParameterKey\": \"output_one\"\n",
      "                      }\n",
      "                    },\n",
      "                    \"text3\": {\n",
      "                      \"taskOutputParameter\": {\n",
      "                        \"producerTask\": \"two-outputs\",\n",
      "                        \"outputParameterKey\": \"output_two\"\n",
      "                      }\n",
      "                    }\n",
      "                  }\n",
      "                },\n",
      "                \"dependentTasks\": [\n",
      "                  \"hello-world\",\n",
      "                  \"two-outputs\"\n",
      "                ],\n",
      "                \"cachingOptions\": {\n",
      "                  \"enableCache\": true\n",
      "                },\n",
      "                \"componentRef\": {\n",
      "                  \"name\": \"comp-consumer\"\n",
      "                }\n",
      "              },\n",
      "              \"hello-world\": {\n",
      "                \"taskInfo\": {\n",
      "                  \"name\": \"hello-world\"\n",
      "                },\n",
      "                \"inputs\": {\n",
      "                  \"parameters\": {\n",
      "                    \"text\": {\n",
      "                      \"componentInputParameter\": \"text\"\n",
      "                    }\n",
      "                  }\n",
      "                },\n",
      "                \"cachingOptions\": {\n",
      "                  \"enableCache\": true\n",
      "                },\n",
      "                \"componentRef\": {\n",
      "                  \"name\": \"comp-hello-world\"\n",
      "                }\n",
      "              },\n",
      "              \"two-outputs\": {\n",
      "                \"taskInfo\": {\n",
      "                  \"name\": \"two-outputs\"\n",
      "                },\n",
      "                \"inputs\": {\n",
      "                  \"parameters\": {\n",
      "                    \"text\": {\n",
      "                      \"componentInputParameter\": \"text\"\n",
      "                    }\n",
      "                  }\n",
      "                },\n",
      "                \"cachingOptions\": {\n",
      "                  \"enableCache\": true\n",
      "                },\n",
      "                \"componentRef\": {\n",
      "                  \"name\": \"comp-two-outputs\"\n",
      "                }\n",
      "              }\n",
      "            }\n",
      "          }\n",
      "        },\n",
      "        \"sdkVersion\": \"kfp-1.8.14\",\n",
      "        \"pipelineInfo\": {\n",
      "          \"name\": \"hello-world-v2\"\n",
      "        },\n",
      "        \"deploymentSpec\": {\n",
      "          \"executors\": {\n",
      "            \"exec-consumer\": {\n",
      "              \"container\": {\n",
      "                \"image\": \"python:3.7\",\n",
      "                \"command\": [\n",
      "                  \"sh\",\n",
      "                  \"-c\",\n",
      "                  \"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet     --no-warn-script-location 'kfp==1.8.14' && \\\"$0\\\" \\\"$@\\\"\\n\",\n",
      "                  \"sh\",\n",
      "                  \"-ec\",\n",
      "                  \"program_path=$(mktemp -d)\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\npython3 -m kfp.v2.components.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\n",
      "                  \"\\nimport kfp\\nfrom kfp.v2 import dsl\\nfrom kfp.v2.dsl import *\\nfrom typing import *\\n\\ndef consumer(text1: str, text2: str, text3: str):\\n    print(f\\\"text1: {text1}; text2: {text2}; text3: {text3}\\\")\\n\\n\"\n",
      "                ],\n",
      "                \"args\": [\n",
      "                  \"--executor_input\",\n",
      "                  \"{{$}}\",\n",
      "                  \"--function_to_execute\",\n",
      "                  \"consumer\"\n",
      "                ]\n",
      "              }\n",
      "            },\n",
      "            \"exec-hello-world\": {\n",
      "              \"container\": {\n",
      "                \"image\": \"python:3.9\",\n",
      "                \"command\": [\n",
      "                  \"sh\",\n",
      "                  \"-c\",\n",
      "                  \"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet     --no-warn-script-location 'kfp==1.8.14' && \\\"$0\\\" \\\"$@\\\"\\n\",\n",
      "                  \"sh\",\n",
      "                  \"-ec\",\n",
      "                  \"program_path=$(mktemp -d)\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\npython3 -m kfp.v2.components.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\n",
      "                  \"\\nimport kfp\\nfrom kfp.v2 import dsl\\nfrom kfp.v2.dsl import *\\nfrom typing import *\\n\\ndef hello_world(text: str) -\\u003e str:\\n    print(text)\\n    return text\\n\\n\"\n",
      "                ],\n",
      "                \"args\": [\n",
      "                  \"--executor_input\",\n",
      "                  \"{{$}}\",\n",
      "                  \"--function_to_execute\",\n",
      "                  \"hello_world\"\n",
      "                ]\n",
      "              }\n",
      "            },\n",
      "            \"exec-two-outputs\": {\n",
      "              \"container\": {\n",
      "                \"image\": \"python:3.7\",\n",
      "                \"command\": [\n",
      "                  \"sh\",\n",
      "                  \"-c\",\n",
      "                  \"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet     --no-warn-script-location 'google-cloud-storage' 'kfp==1.8.14' && \\\"$0\\\" \\\"$@\\\"\\n\",\n",
      "                  \"sh\",\n",
      "                  \"-ec\",\n",
      "                  \"program_path=$(mktemp -d)\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\npython3 -m kfp.v2.components.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\n",
      "                  \"\\nimport kfp\\nfrom kfp.v2 import dsl\\nfrom kfp.v2.dsl import *\\nfrom typing import *\\n\\ndef two_outputs(\\n    text: str,\\n) -\\u003e NamedTuple(\\n    \\\"Outputs\\\",\\n    [\\n        (\\\"output_one\\\", str),  # Return parameters\\n        (\\\"output_two\\\", str),\\n    ],\\n):\\n    # the import is not actually used for this simple example, but the import\\n    # is successful, as it was included in the `packages_to_install` list.\\n    from google.cloud import storage  # noqa: F401\\n\\n    o1 = f\\\"output one from text: {text}\\\"\\n    o2 = f\\\"output two from text: {text}\\\"\\n    print(\\\"output one: {}; output_two: {}\\\".format(o1, o2))\\n    return (o1, o2)\\n\\n\"\n",
      "                ],\n",
      "                \"args\": [\n",
      "                  \"--executor_input\",\n",
      "                  \"{{$}}\",\n",
      "                  \"--function_to_execute\",\n",
      "                  \"two_outputs\"\n",
      "                ]\n",
      "              }\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "      },\n",
      "      \"state\": \"PIPELINE_STATE_SUCCEEDED\",\n",
      "      \"jobDetail\": {\n",
      "        \"pipelineContext\": {\n",
      "          \"name\": \"projects/192756378758/locations/us-central1/metadataStores/default/contexts/hello-world-v2\"\n",
      "        },\n",
      "        \"pipelineRunContext\": {\n",
      "          \"name\": \"projects/192756378758/locations/us-central1/metadataStores/default/contexts/184807013908217856\"\n",
      "        }\n",
      "      },\n",
      "      \"runtimeConfig\": {\n",
      "        \"parameters\": {\n",
      "          \"text\": {\n",
      "            \"stringValue\": \"hi there\"\n",
      "          }\n",
      "        },\n",
      "        \"gcsOutputDirectory\": \"gs://qwiklabs-gcp-01-d69e76763702/pipeline_root/intro\"\n",
      "      },\n",
      "      \"serviceAccount\": \"192756378758-compute@developer.gserviceaccount.com\"\n",
      "    },\n",
      "    {\n",
      "      \"name\": \"projects/192756378758/locations/us-central1/pipelineJobs/hello-world-v2-20220922143209\",\n",
      "      \"displayName\": \"hello-world-v2-20220922143209\",\n",
      "      \"createTime\": \"2022-09-22T14:32:09.858920Z\",\n",
      "      \"startTime\": \"2022-09-22T14:32:10.420175Z\",\n",
      "      \"endTime\": \"2022-09-22T14:32:12.894671Z\",\n",
      "      \"updateTime\": \"2022-09-22T14:32:12.894671Z\",\n",
      "      \"pipelineSpec\": {\n",
      "        \"deploymentConfig\": {\n",
      "          \"@type\": \"type.googleapis.com/ml_pipelines.PipelineDeploymentConfig\",\n",
      "          \"executors\": {\n",
      "            \"exec-consumer\": {\n",
      "              \"container\": {\n",
      "                \"image\": \"python:3.7\",\n",
      "                \"command\": [\n",
      "                  \"sh\",\n",
      "                  \"-c\",\n",
      "                  \"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet     --no-warn-script-location 'kfp==1.8.14' && \\\"$0\\\" \\\"$@\\\"\\n\",\n",
      "                  \"sh\",\n",
      "                  \"-ec\",\n",
      "                  \"program_path=$(mktemp -d)\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\npython3 -m kfp.v2.components.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\n",
      "                  \"\\nimport kfp\\nfrom kfp.v2 import dsl\\nfrom kfp.v2.dsl import *\\nfrom typing import *\\n\\ndef consumer(text1: str, text2: str, text3: str):\\n    print(f\\\"text1: {text1}; text2: {text2}; text3: {text3}\\\")\\n\\n\"\n",
      "                ],\n",
      "                \"args\": [\n",
      "                  \"--executor_input\",\n",
      "                  \"{{$}}\",\n",
      "                  \"--function_to_execute\",\n",
      "                  \"consumer\"\n",
      "                ]\n",
      "              }\n",
      "            },\n",
      "            \"exec-hello-world\": {\n",
      "              \"container\": {\n",
      "                \"image\": \"python:3.9\",\n",
      "                \"command\": [\n",
      "                  \"sh\",\n",
      "                  \"-c\",\n",
      "                  \"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet     --no-warn-script-location 'kfp==1.8.14' && \\\"$0\\\" \\\"$@\\\"\\n\",\n",
      "                  \"sh\",\n",
      "                  \"-ec\",\n",
      "                  \"program_path=$(mktemp -d)\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\npython3 -m kfp.v2.components.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\n",
      "                  \"\\nimport kfp\\nfrom kfp.v2 import dsl\\nfrom kfp.v2.dsl import *\\nfrom typing import *\\n\\ndef hello_world(text: str) -\\u003e str:\\n    print(text)\\n    return text\\n\\n\"\n",
      "                ],\n",
      "                \"args\": [\n",
      "                  \"--executor_input\",\n",
      "                  \"{{$}}\",\n",
      "                  \"--function_to_execute\",\n",
      "                  \"hello_world\"\n",
      "                ]\n",
      "              }\n",
      "            },\n",
      "            \"exec-two-outputs\": {\n",
      "              \"container\": {\n",
      "                \"image\": \"python:3.7\",\n",
      "                \"command\": [\n",
      "                  \"sh\",\n",
      "                  \"-c\",\n",
      "                  \"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet     --no-warn-script-location 'google-cloud-storage' 'kfp==1.8.14' && \\\"$0\\\" \\\"$@\\\"\\n\",\n",
      "                  \"sh\",\n",
      "                  \"-ec\",\n",
      "                  \"program_path=$(mktemp -d)\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\npython3 -m kfp.v2.components.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\n",
      "                  \"\\nimport kfp\\nfrom kfp.v2 import dsl\\nfrom kfp.v2.dsl import *\\nfrom typing import *\\n\\ndef two_outputs(\\n    text: str,\\n) -\\u003e NamedTuple(\\n    \\\"Outputs\\\",\\n    [\\n        (\\\"output_one\\\", str),  # Return parameters\\n        (\\\"output_two\\\", str),\\n    ],\\n):\\n    # the import is not actually used for this simple example, but the import\\n    # is successful, as it was included in the `packages_to_install` list.\\n    from google.cloud import storage  # noqa: F401\\n\\n    o1 = f\\\"output one from text: {text}\\\"\\n    o2 = f\\\"output two from text: {text}\\\"\\n    print(\\\"output one: {}; output_two: {}\\\".format(o1, o2))\\n    return (o1, o2)\\n\\n\"\n",
      "                ],\n",
      "                \"args\": [\n",
      "                  \"--executor_input\",\n",
      "                  \"{{$}}\",\n",
      "                  \"--function_to_execute\",\n",
      "                  \"two_outputs\"\n",
      "                ]\n",
      "              }\n",
      "            }\n",
      "          }\n",
      "        },\n",
      "        \"components\": {\n",
      "          \"comp-consumer\": {\n",
      "            \"inputDefinitions\": {\n",
      "              \"parameters\": {\n",
      "                \"text1\": {\n",
      "                  \"type\": \"STRING\"\n",
      "                },\n",
      "                \"text2\": {\n",
      "                  \"type\": \"STRING\"\n",
      "                },\n",
      "                \"text3\": {\n",
      "                  \"type\": \"STRING\"\n",
      "                }\n",
      "              }\n",
      "            },\n",
      "            \"executorLabel\": \"exec-consumer\"\n",
      "          },\n",
      "          \"comp-hello-world\": {\n",
      "            \"inputDefinitions\": {\n",
      "              \"parameters\": {\n",
      "                \"text\": {\n",
      "                  \"type\": \"STRING\"\n",
      "                }\n",
      "              }\n",
      "            },\n",
      "            \"outputDefinitions\": {\n",
      "              \"parameters\": {\n",
      "                \"Output\": {\n",
      "                  \"type\": \"STRING\"\n",
      "                }\n",
      "              }\n",
      "            },\n",
      "            \"executorLabel\": \"exec-hello-world\"\n",
      "          },\n",
      "          \"comp-two-outputs\": {\n",
      "            \"inputDefinitions\": {\n",
      "              \"parameters\": {\n",
      "                \"text\": {\n",
      "                  \"type\": \"STRING\"\n",
      "                }\n",
      "              }\n",
      "            },\n",
      "            \"outputDefinitions\": {\n",
      "              \"parameters\": {\n",
      "                \"output_one\": {\n",
      "                  \"type\": \"STRING\"\n",
      "                },\n",
      "                \"output_two\": {\n",
      "                  \"type\": \"STRING\"\n",
      "                }\n",
      "              }\n",
      "            },\n",
      "            \"executorLabel\": \"exec-two-outputs\"\n",
      "          }\n",
      "        },\n",
      "        \"schemaVersion\": \"2.0.0\",\n",
      "        \"root\": {\n",
      "          \"inputDefinitions\": {\n",
      "            \"parameters\": {\n",
      "              \"text\": {\n",
      "                \"type\": \"STRING\"\n",
      "              }\n",
      "            }\n",
      "          },\n",
      "          \"dag\": {\n",
      "            \"tasks\": {\n",
      "              \"consumer\": {\n",
      "                \"taskInfo\": {\n",
      "                  \"name\": \"consumer\"\n",
      "                },\n",
      "                \"inputs\": {\n",
      "                  \"parameters\": {\n",
      "                    \"text1\": {\n",
      "                      \"taskOutputParameter\": {\n",
      "                        \"producerTask\": \"hello-world\",\n",
      "                        \"outputParameterKey\": \"Output\"\n",
      "                      }\n",
      "                    },\n",
      "                    \"text2\": {\n",
      "                      \"taskOutputParameter\": {\n",
      "                        \"producerTask\": \"two-outputs\",\n",
      "                        \"outputParameterKey\": \"output_one\"\n",
      "                      }\n",
      "                    },\n",
      "                    \"text3\": {\n",
      "                      \"taskOutputParameter\": {\n",
      "                        \"producerTask\": \"two-outputs\",\n",
      "                        \"outputParameterKey\": \"output_two\"\n",
      "                      }\n",
      "                    }\n",
      "                  }\n",
      "                },\n",
      "                \"dependentTasks\": [\n",
      "                  \"hello-world\",\n",
      "                  \"two-outputs\"\n",
      "                ],\n",
      "                \"cachingOptions\": {\n",
      "                  \"enableCache\": true\n",
      "                },\n",
      "                \"componentRef\": {\n",
      "                  \"name\": \"comp-consumer\"\n",
      "                }\n",
      "              },\n",
      "              \"hello-world\": {\n",
      "                \"taskInfo\": {\n",
      "                  \"name\": \"hello-world\"\n",
      "                },\n",
      "                \"inputs\": {\n",
      "                  \"parameters\": {\n",
      "                    \"text\": {\n",
      "                      \"componentInputParameter\": \"text\"\n",
      "                    }\n",
      "                  }\n",
      "                },\n",
      "                \"cachingOptions\": {\n",
      "                  \"enableCache\": true\n",
      "                },\n",
      "                \"componentRef\": {\n",
      "                  \"name\": \"comp-hello-world\"\n",
      "                }\n",
      "              },\n",
      "              \"two-outputs\": {\n",
      "                \"taskInfo\": {\n",
      "                  \"name\": \"two-outputs\"\n",
      "                },\n",
      "                \"inputs\": {\n",
      "                  \"parameters\": {\n",
      "                    \"text\": {\n",
      "                      \"componentInputParameter\": \"text\"\n",
      "                    }\n",
      "                  }\n",
      "                },\n",
      "                \"cachingOptions\": {\n",
      "                  \"enableCache\": true\n",
      "                },\n",
      "                \"componentRef\": {\n",
      "                  \"name\": \"comp-two-outputs\"\n",
      "                }\n",
      "              }\n",
      "            }\n",
      "          }\n",
      "        },\n",
      "        \"sdkVersion\": \"kfp-1.8.14\",\n",
      "        \"pipelineInfo\": {\n",
      "          \"name\": \"hello-world-v2\"\n",
      "        },\n",
      "        \"deploymentSpec\": {\n",
      "          \"executors\": {\n",
      "            \"exec-consumer\": {\n",
      "              \"container\": {\n",
      "                \"image\": \"python:3.7\",\n",
      "                \"command\": [\n",
      "                  \"sh\",\n",
      "                  \"-c\",\n",
      "                  \"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet     --no-warn-script-location 'kfp==1.8.14' && \\\"$0\\\" \\\"$@\\\"\\n\",\n",
      "                  \"sh\",\n",
      "                  \"-ec\",\n",
      "                  \"program_path=$(mktemp -d)\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\npython3 -m kfp.v2.components.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\n",
      "                  \"\\nimport kfp\\nfrom kfp.v2 import dsl\\nfrom kfp.v2.dsl import *\\nfrom typing import *\\n\\ndef consumer(text1: str, text2: str, text3: str):\\n    print(f\\\"text1: {text1}; text2: {text2}; text3: {text3}\\\")\\n\\n\"\n",
      "                ],\n",
      "                \"args\": [\n",
      "                  \"--executor_input\",\n",
      "                  \"{{$}}\",\n",
      "                  \"--function_to_execute\",\n",
      "                  \"consumer\"\n",
      "                ]\n",
      "              }\n",
      "            },\n",
      "            \"exec-hello-world\": {\n",
      "              \"container\": {\n",
      "                \"image\": \"python:3.9\",\n",
      "                \"command\": [\n",
      "                  \"sh\",\n",
      "                  \"-c\",\n",
      "                  \"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet     --no-warn-script-location 'kfp==1.8.14' && \\\"$0\\\" \\\"$@\\\"\\n\",\n",
      "                  \"sh\",\n",
      "                  \"-ec\",\n",
      "                  \"program_path=$(mktemp -d)\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\npython3 -m kfp.v2.components.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\n",
      "                  \"\\nimport kfp\\nfrom kfp.v2 import dsl\\nfrom kfp.v2.dsl import *\\nfrom typing import *\\n\\ndef hello_world(text: str) -\\u003e str:\\n    print(text)\\n    return text\\n\\n\"\n",
      "                ],\n",
      "                \"args\": [\n",
      "                  \"--executor_input\",\n",
      "                  \"{{$}}\",\n",
      "                  \"--function_to_execute\",\n",
      "                  \"hello_world\"\n",
      "                ]\n",
      "              }\n",
      "            },\n",
      "            \"exec-two-outputs\": {\n",
      "              \"container\": {\n",
      "                \"image\": \"python:3.7\",\n",
      "                \"command\": [\n",
      "                  \"sh\",\n",
      "                  \"-c\",\n",
      "                  \"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet     --no-warn-script-location 'google-cloud-storage' 'kfp==1.8.14' && \\\"$0\\\" \\\"$@\\\"\\n\",\n",
      "                  \"sh\",\n",
      "                  \"-ec\",\n",
      "                  \"program_path=$(mktemp -d)\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\npython3 -m kfp.v2.components.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\n",
      "                  \"\\nimport kfp\\nfrom kfp.v2 import dsl\\nfrom kfp.v2.dsl import *\\nfrom typing import *\\n\\ndef two_outputs(\\n    text: str,\\n) -\\u003e NamedTuple(\\n    \\\"Outputs\\\",\\n    [\\n        (\\\"output_one\\\", str),  # Return parameters\\n        (\\\"output_two\\\", str),\\n    ],\\n):\\n    # the import is not actually used for this simple example, but the import\\n    # is successful, as it was included in the `packages_to_install` list.\\n    from google.cloud import storage  # noqa: F401\\n\\n    o1 = f\\\"output one from text: {text}\\\"\\n    o2 = f\\\"output two from text: {text}\\\"\\n    print(\\\"output one: {}; output_two: {}\\\".format(o1, o2))\\n    return (o1, o2)\\n\\n\"\n",
      "                ],\n",
      "                \"args\": [\n",
      "                  \"--executor_input\",\n",
      "                  \"{{$}}\",\n",
      "                  \"--function_to_execute\",\n",
      "                  \"two_outputs\"\n",
      "                ]\n",
      "              }\n",
      "            }\n",
      "          }\n",
      "        }\n",
      "      },\n",
      "      \"state\": \"PIPELINE_STATE_SUCCEEDED\",\n",
      "      \"jobDetail\": {\n",
      "        \"pipelineContext\": {\n",
      "          \"name\": \"projects/192756378758/locations/us-central1/metadataStores/default/contexts/hello-world-v2\"\n",
      "        },\n",
      "        \"pipelineRunContext\": {\n",
      "          \"name\": \"projects/192756378758/locations/us-central1/metadataStores/default/contexts/hello-world-v2-20220922143209\"\n",
      "        }\n",
      "      },\n",
      "      \"runtimeConfig\": {\n",
      "        \"parameters\": {\n",
      "          \"text\": {\n",
      "            \"stringValue\": \"hi there\"\n",
      "          }\n",
      "        },\n",
      "        \"gcsOutputDirectory\": \"gs://qwiklabs-gcp-01-d69e76763702/pipeline_root/intro\"\n",
      "      },\n",
      "      \"serviceAccount\": \"192756378758-compute@developer.gserviceaccount.com\"\n",
      "    }\n",
      "  ]\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "! curl -X GET -H \"Authorization: Bearer $(gcloud auth print-access-token)\"   -H \"Content-Type: application/json\"   https://{API_ENDPOINT}/v1beta1/projects/{PROJECT_ID}/locations/{REGION}/pipelineJobs/{PIPELINE_RUN_ID}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "pipelines_rest_cancel"
   },
   "source": [
    "### Cancel a pipeline job given its ID"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "id": "YlJeXYBK2Ou_"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<!DOCTYPE html>\n",
      "<html lang=en>\n",
      "  <meta charset=utf-8>\n",
      "  <meta name=viewport content=\"initial-scale=1, minimum-scale=1, width=device-width\">\n",
      "  <title>Error 404 (Not Found)!!1</title>\n",
      "  <style>\n",
      "    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}\n",
      "  </style>\n",
      "  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>\n",
      "  <p><b>404.</b> <ins>That’s an error.</ins>\n",
      "  <p>The requested URL <code>/v1beta1/projects/qwiklabs-gcp-01-d69e76763702/locations/us-central1/pipelineJobs/</code> was not found on this server.  <ins>That’s all we know.</ins>\n"
     ]
    }
   ],
   "source": [
    "! curl -X DELETE -H \"Authorization: Bearer $(gcloud auth print-access-token)\"   -H \"Content-Type: application/json\"   https://{API_ENDPOINT}/v1beta1/projects/{PROJECT_ID}/locations/{REGION}/pipelineJobs/{PIPELINE_RUN_ID}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "pipelines_rest_delete"
   },
   "source": [
    "### Delete a pipeline job given its ID"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "id": "WIprXZMc2Ou_"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<!DOCTYPE html>\n",
      "<html lang=en>\n",
      "  <meta charset=utf-8>\n",
      "  <meta name=viewport content=\"initial-scale=1, minimum-scale=1, width=device-width\">\n",
      "  <title>Error 404 (Not Found)!!1</title>\n",
      "  <style>\n",
      "    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}\n",
      "  </style>\n",
      "  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>\n",
      "  <p><b>404.</b> <ins>That’s an error.</ins>\n",
      "  <p>The requested URL <code>/v1beta1/projects/qwiklabs-gcp-01-d69e76763702/locations/us-central1/pipelineJobs/</code> was not found on this server.  <ins>That’s all we know.</ins>\n"
     ]
    }
   ],
   "source": [
    "! curl -X DELETE -H \"Authorization: Bearer $(gcloud auth print-access-token)\"   -H \"Content-Type: application/json\"   https://{API_ENDPOINT}/v1beta1/projects/{PROJECT_ID}/locations/{REGION}/pipelineJobs/{PIPELINE_RUN_ID}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "cleanup:pipelines"
   },
   "source": [
    "# Cleaning up\n",
    "\n",
    "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
    "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
    "\n",
    "Otherwise, you can delete the individual resources you created in this tutorial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "id": "-eXdbQCo2Ou_"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "list index out of range\n",
      "Removing gs://qwiklabs-gcp-01-d69e76763702/pipeline_root/#1663856036655692...\n",
      "Removing gs://qwiklabs-gcp-01-d69e76763702/pipeline_root/intro/#1663856036809239...\n",
      "Removing gs://qwiklabs-gcp-01-d69e76763702/pipeline_root/intro/192756378758/#1663856036963636...\n",
      "Removing gs://qwiklabs-gcp-01-d69e76763702/pipeline_root/intro/192756378758/hello-world-v2-20220922141223/#1663856037139451...\n",
      "/ [4 objects]                                                                   \n",
      "==> NOTE: You are performing a sequence of gsutil operations that may\n",
      "run significantly faster if you instead use gsutil -m rm ... Please\n",
      "see the -m section under \"gsutil help options\" for further information\n",
      "about when gsutil -m can be advantageous.\n",
      "\n",
      "Removing gs://qwiklabs-gcp-01-d69e76763702/pipeline_root/intro/192756378758/hello-world-v2-20220922141223/consumer_5276821833755131904/#1663856147426088...\n",
      "Removing gs://qwiklabs-gcp-01-d69e76763702/pipeline_root/intro/192756378758/hello-world-v2-20220922141223/consumer_5276821833755131904/executor_output.json#1663856147782828...\n",
      "Removing gs://qwiklabs-gcp-01-d69e76763702/pipeline_root/intro/192756378758/hello-world-v2-20220922141223/hello-world_-3946550203099643904/#1663856040813854...\n",
      "Removing gs://qwiklabs-gcp-01-d69e76763702/pipeline_root/intro/192756378758/hello-world-v2-20220922141223/hello-world_-3946550203099643904/executor_output.json#1663856041096954...\n",
      "Removing gs://qwiklabs-gcp-01-d69e76763702/pipeline_root/intro/192756378758/hello-world-v2-20220922141223/two-outputs_2970978824541437952/#1663856037351992...\n",
      "Removing gs://qwiklabs-gcp-01-d69e76763702/pipeline_root/intro/192756378758/hello-world-v2-20220922141223/two-outputs_2970978824541437952/executor_output.json#1663856037641307...\n",
      "Removing gs://qwiklabs-gcp-01-d69e76763702/pipeline_root/intro/192756378758/hello-world-v2-20220922143250/#1663857282916332...\n",
      "Removing gs://qwiklabs-gcp-01-d69e76763702/pipeline_root/intro/192756378758/hello-world-v2-20220922143250/consumer_2100658196552089600/#1663857398844721...\n",
      "Removing gs://qwiklabs-gcp-01-d69e76763702/pipeline_root/intro/192756378758/hello-world-v2-20220922143250/consumer_2100658196552089600/executor_output.json#1663857399137702...\n",
      "Removing gs://qwiklabs-gcp-01-d69e76763702/pipeline_root/intro/192756378758/hello-world-v2-20220922143250/hello-world_-7122713840302686208/#1663857283626026...\n",
      "Removing gs://qwiklabs-gcp-01-d69e76763702/pipeline_root/intro/192756378758/hello-world-v2-20220922143250/hello-world_-7122713840302686208/executor_output.json#1663857283916648...\n",
      "Removing gs://qwiklabs-gcp-01-d69e76763702/pipeline_root/intro/192756378758/hello-world-v2-20220922143250/two-outputs_6712344214979477504/#1663857283115641...\n",
      "Removing gs://qwiklabs-gcp-01-d69e76763702/pipeline_root/intro/192756378758/hello-world-v2-20220922143250/two-outputs_6712344214979477504/executor_output.json#1663857283490946...\n",
      "/ [17 objects]                                                                  \n",
      "Operation completed over 17 objects.                                             \n",
      "Removing gs://qwiklabs-gcp-01-d69e76763702/...\n"
     ]
    }
   ],
   "source": [
    "delete_pipeline = True\n",
    "delete_bucket = True\n",
    "\n",
    "try:\n",
    "    if delete_pipeline and \"DISPLAY_NAME\" in globals():\n",
    "        pipelines = aip.PipelineJob.list(\n",
    "            filter=f\"display_name={DISPLAY_NAME}\", order_by=\"create_time\"\n",
    "        )\n",
    "        pipeline = pipelines[0]\n",
    "        aip.PipelineJob.delete(pipeline.resource_name)\n",
    "        print(\"Deleted pipeline:\", pipeline)\n",
    "except Exception as e:\n",
    "    print(e)\n",
    "\n",
    "if delete_bucket or os.getenv(\"IS_TESTING\"):\n",
    "    ! gsutil rm -r $BUCKET_URI"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "name": "pipelines_intro_kfp.ipynb",
   "toc_visible": true
  },
  "environment": {
   "kernel": "python3",
   "name": "tf2-gpu.2-6.m96",
   "type": "gcloud",
   "uri": "gcr.io/deeplearning-platform-release/tf2-gpu.2-6:m96"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
